Collaborative Web Crawler over High-speed Research Network

This paper proposes an idea for constructing a distributed web crawler by utilizing existing high-speed research networks. This is an initial effort of the Web Language Engineering (WLE) project which investigates techniques in processing the languages found in published web documents. In this paper, we focus on designing a geographically distributed web crawler. Multiple crawlers work collaboratively and use an existing research network for communication. Two methods for distributing the workload of web crawling are investigated. The first one divides the workload among these crawlers by selecting the nearest crawler to retrieve each domain, while the second one distributes the crawling task by using a hash function. As a prototype, we implement two crawlers, one is located in Thailand while another one is located in Japan. Some statistics about the prototype crawler are shown. In the end, we discuss about the use of collaborative crawling for the real-time monitoring of language usage in web documents.